Automated Protocol Analysis : Tools and Methodology TR 91 - 034 August , 1991

The Textlab Research Group, over the past seven years, has developed a collection of tools and techniques for recording users' interactions with computer systems in machine-readable form and for automatically analyzing and displaying those data. This report catalogs those tools and discusses their methodological context and implications. Tools discussed include the following: tracking users' behaviors and producing a machine-recorded protocol at the level of action -data an order of magnitude larger than keystrokes; replaying users' sessions from protocol data; modeling users' strategies using formal cognitive grammars; analyzing user sessions by parsing them with the grammars; and displaying results in visual form -both static and animated to facilitate interpretation and understanding by researchers. These tools are placed in a methodological context by reviewing issues associated with concurrent think-aloud, keystroke, and video protocols; and other computer systems are reviewed that support these forms of protocol data. The discussion concludes by noting the increased importance of data management required by automated methods and our thoughts for a comprehensive environment, built around a protocol management facility, that would integrate the tools discussed and support development of new ones.